Data monitoring and management device and event data monitoring method

ABSTRACT

According to one embodiment, a device includes an instruction unit which records in a recording medium, event-related data of when an event is detected and monitoring data of when the event occurs, and a display data output unit which outputs from the recording medium and plays as display data, the event-related data and a part of the monitoring data corresponding to the event-related data. If there is a specification input to the displayed event-related data, the monitoring data corresponding to the event-related data is played.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.15/445,856, filed Feb. 28, 2017 which is based upon and claims thebenefit of priority from Japanese Patent Application No. 2016-200993,filed Oct. 12, 2016, the entire contents of which are incorporatedherein by reference.

FIELD

Embodiments described herein relate generally to a data monitoring andmanagement device and an event data monitoring method.

BACKGROUND

There are monitors which record video data and/or audio data inrecorders. These monitors are used in houses, schools, hospitals,airports, shopping malls, and various other facilities. Further, themonitors are also used as voice recorders in airplanes, drive recordersin taxis, and the like. Still further, some monitors can transfer videodata from monitoring cameras to portable devices.

BRIEF DESCRIPTION OF THE DRAWINGS

A general architecture that implements the various features of theembodiments will now be described with reference to the drawings. Thedrawings and the associated descriptions are provided to illustrate theembodiments and not to limit the scope of the invention.

FIG. 1 is a diagram showing an example of the overall structure of anetwork system which adopts a data monitoring and management device ofan embodiment.

FIG. 2 is an explanatory diagram showing on a timeline, an example of astream of event-related data and a stream of monitoring data recorded inthe embodiment of FIG. 1.

FIG. 3 is a block diagram showing main structural elements of theembodiment.

FIG. 4 is a diagram showing an example of a menu screen of a smartphoneas a user interface which allows the user to access the event-relateddata and/or the monitoring data.

FIG. 5A is an explanatory diagram showing an example of the procedurefor accessing the monitoring data from the smartphone.

FIG. 5B is an explanatory diagram showing another example of theprocedure for accessing the monitoring data from the smartphone.

FIG. 6A is a diagram showing an example of the operation screen of thesmartphone.

FIG. 6B is a diagram showing another example of the operation screen ofthe smartphone.

FIG. 7 is a diagram showing an example of video to be played whenmonitoring data (video data) related to certain events is played.

FIG. 8A is an explanatory diagram showing an example of the relationshipof the smartphone to the event-related data displayed on the screen ofthe smartphone and the operation method of the event-related datadisplayed on the screen of the smartphone.

FIG. 8B is an explanatory diagram showing another example of therelationship of the smartphone to the event-related data displayed onthe screen of the smartphone and the operation method of theevent-related data displayed on the screen of the smartphone.

FIG. 9 is an explanatory diagram showing another example of therelationship of the smartphone to the event-related data displayed onthe screen of the smartphone and the operation method of theevent-related data displayed on the screen of the smartphone.

FIG. 10 is a hierarchy chart showing an example of the recordingposition relationship between the event-related data and the monitoringdata.

FIG. 11 is a diagram showing an example of the overall structure of anetwork system which adopts another embodiment.

FIG. 12 is a diagram showing an example of the embodiment where a listof sensor operation modes is displayed.

FIG. 13 is a diagram showing an example of the embodiment where agateway performs communication with a plurality of sensors using aplurality of wireless communication standards.

FIG. 14A is a diagram showing the first-type home gateway.

FIG. 14B is a diagram showing the second-type home gateway.

FIG. 14C is a diagram showing the third-type home gateway.

FIG. 15A is a diagram showing an example of a switch sensor.

FIG. 15B is a diagram showing an example of a light sensor.

FIG. 16 is a diagram showing the other embodiment.

FIG. 17 is a diagram showing an example of the embodiment of FIG. 16where a list of sensor operation modes is displayed.

DETAILED DESCRIPTION

Various embodiments will be described hereinafter with reference to theaccompanying drawings.

It is preferable that monitors which record monitoring data such asvideo data and/or audio data should allow the users to effectively checkthe recorded data.

In general, according to one embodiment, there are provided a datamonitoring and management device which records, based on detection of atleast one event by a sensor, monitoring data and event-related datawhich is related to the event, uses the event-related data effectively,and allows the user to check the recorded monitoring data furthereffectively.

According to one embodiment, a data monitoring and management devicecomprises an instruction unit (621) which records event-related data ofwhen an event is detected and monitoring data of when the event occursin a recording medium; and a display data output unit (627) whichoutputs the event-related data from the recording medium and plays theevent-related data as display data, wherein the display data output unitoutputs, when receiving a specification input to the displayedevent-related data, monitoring data which corresponds to theevent-related data.

An embodiment will further be described with reference to the drawings.

In FIG. 1, a server 1000 is connectable to a home gateway (hereinafterreferred to as an HGW) 600 via the Internet 300. The HGW 600 comprises acontrol data management unit 601, a system controller 602, a devicemanagement unit 603, a network interface (hereinafter referred to as anetwork I/F) 605, a recording management unit 607, a camera 611, amicrophone 613, a speaker 615, and the like. Further, the HGW 600includes a sensor control table 906.

The control data manager 601 includes an application management unit(hereinafter referred to as an APP-Mg), an event management unit(hereinafter referred to as an EVT-Mg), and a configuration managementunit (hereinafter referred to as a CONFIG-Mg). The APP-Mg manages aplurality of applications which control various operations of the HGW600. The EVT-Mg manages an event application which controls variousoperations resulting from occurrence of various events which will bedescribed later. Further, the CONF-Mg recognizes functions providedwithin the HGW 600 and various other functions related to the HGW 600and manages a configuration application which controls an operationorder, imposes an operation restriction, and the like, for example.

The system controller 602 can perform sequence control by controllingall the function blocks of the HGW 600.

The EVT-Mg can control the camera 611, the microphone 613, the speaker615, the recording management unit 607, and the like. Further, theEVT-Mg can determine data which is detected from an external sensor andis input from the network I/F 605 and/or data which is obtained from thecamera 611, the microphone 613, and the speaker 615, and can control thenext actions or operations. The CONFG-Mg can make an initial setting ofeach function block of the HGW 600, and various other settings such as afunction restriction setting, a function expansion setting, a priorityorder setting, and an operation time setting.

The device management unit 603 can approve other devices which operatein association with the HGW 600 and can register these devices in amemory. Therefore, the device management unit 603 can manage othersensors 101, 102, 103, . . . which are in connection via the network I/F605. Further, the device management unit 603 also registersidentification data of the server 1000 which is connected via theInternet 300, and thus the device management unit 603 can recognize theserver 1000. Still further, the device management unit 603 alsoregisters identification data of a smartphone GUI-1 which is connectedvia the Internet 300 and the like, and thus the device management unit603 can recognize the smartphone GUI-1 and the like.

Still further, the sensor control table 609 stores the name, theposition information, and the control data of each of the sensors 101,102, 103 and 104 which will be described later. Still further, the nameand the position information of each sensor can be displayed on thescreen of the smartphone GUI-1, and thus the user can check the types ofsensor or the installation positions of the sensors.

The network I/F 605 is connected to the other sensors 101, 102, 103, . .. , for example, within a house via Near Field Communication. In thedrawing, the structure of the sensor 101 is representatively shown. Thesensor 101 includes a control data management unit 112 and a network I/F115. Further, the sensor 101 has functions 116 and 117 as sensingelements. Note that the sensor is not necessarily limited to this typebut may be various other types.

The control data management unit 112 includes an application managementunit (APP-Mg), an event management unit (EVT-Mg), and a configurationmanagement unit (CONFIG-Mg). The CONFIG-Mg manages various applicationswhich control the operation of the whole sensor system. The EVT-Mgmanages an event application which performs the next operation of thesensor 101 based on the detected data from the functions 116 and 117.The functions 116 and 117 include structural elements serving theirsensing purposes, respectively. For example, there is a case where acamera and a microphone are provided as the structural elements as inthe HGW 600. Further, the structural elements also include a thermalsensor, a temperature sensor, a humidity sensor, a light sensor, apressure sensor, a switch, and the like. According to the operationpurpose, the sensor 101 may comprise one sensing element or a pluralityof sensing elements.

The above-described sensors 101, 102, 103, . . . are installed invarious positions, for example, in the house as a sensor which detectsan opening-closing movement of a door, a sensor which detects a movementof a person, a sensor which detects an opening-closing movement of awindow, a sensor which shoots a video, or the like.

In the above-described system, when one or more detection signals areoutput from one or more of the sensor (camera) 611, the sensor(microphone) 613, and the other sensors 101, 102, . . . , the controldata management unit 601 recognizes event occurrence. Then, the controldata management unit 601 controls the camera 611 via the recordingmanagement unit 607. In this way, the camera 611 transmits monitoringdata which has been cached since before (for example, since ten minutesbefore) the event occurrence time to a recording medium via therecording management unit 607 and the control data management unit 601,and continuously transmits the recorded monitoring data (video data) fora certain period of time (for example, for three, five, ten, twenty, orthirty minutes). In the present system, along with monitoring data,event-related data (referred to also as event attribute data) of whenthe event is detected is also transmitted to a recording medium 1010.

The event-related data can include, for example, one or more of theevent occurrence time, the sensor which has detected the event, theposition data of the sensor, the recording start time, the recording endtime, and the like.

Note that, although the recording medium 1010 is assumed to be, forexample, a memory in the sever 1000 in FIG. 1, the recording medium 1010is not necessarily a recording medium provided in the sever. Further,the monitoring data may be recorded in a recording medium in the HGW 600or may be recorded in a recording medium which is connected via thenetwork I/F 605. The recording medium 1010 comprises a data area 1011and a management area 1021. Further, monitoring data 1012 is recorded inthe data area 1011, and event-related data 1022 is recorded in themanagement area 1021.

There is a case where the monitoring data 1012 includes video data aswell as measurement data from a sensor. For example, the measurementdata includes a change in temperature, a change in humidity, a change inpressure, and the like in a particular position. In the management area1021, management data used for playing the monitoring data is recorded.The previously-mentioned event-related data is included in themanagement data. The management data includes event-related data and therecording address of monitoring data corresponding to the event-relateddata. In response to a plurality of events, a plurality of items ofevent-related data and a plurality of items of monitoring datacorresponding to the plurality of items of event-related data arerecorded, accordingly.

The event-related data includes the type of event (referred to also asthe type of sensor). Further, in response to an event, monitoring data(for example, monitoring video) is recorded, and event-related dataincludes the recording start time, the recording end time, and the likeof the monitoring data.

FIG. 2 shows the progress of monitor data recording processes in therecording medium with time in response to events. The followingdescription is based on the assumption that various sensors areinstalled in the living room of the house. As the sensors, a sensorwhich detects an opening-closing movement of a door 1, a sensor whichdetects an opening-closing movement of a door 2, a sensor which detectsan opening-closing movement of a window 1, a sensor which detects anopening-closing movement of a window 2, a microphone, a movementdetection sensor (such as a video sensor or an infrared sensor) areinstalled. Further, the HGW 600 is installed in the corner of theceiling of the living room, and the camera can shoot a video of theliving room.

Now, a child 1 enters the living room from the door 1, and the door 1 isopened at a time t1 and is closed at a time t2. At the time t1,movements are detected by the camera. When a door is opened or closed, avideo is continuously recorded for about three minutes, for example. Inthis detection period, movements are continuously detected, and a videois continuously recorded. While the video is recorded, sounds are pickedby the microphone 613. Consequently, the monitoring data resulting fromthe first event (note that, in this case, two events have occurred) isrecorded as recording data Rec1, for example, in the recording medium1010 (or may be a recording medium in the HGW 600 or a recording mediumwhich is directly connected to the HGW 600, if any). Further, as theevent-related data, the sensor ID of the sensor which is attached to thedoor 1, the camera ID of the camera 611, and the recording start timeand the recording end time of the recording data Rec1. Still further,the management data (event-related data) also includes an address in therecording medium 1010 at which the recording data Rec1 is recorded.

A short time later, a child 2 enters the living room from the door 2,and the door 2 is opened at a time t3 and is closed at a time t4. At thetime t3, movements are detected by the camera. The monitoring dataresulting from the second event (note that, in this case, two eventshave occurred) is recorded as recording data Rec2 in the recordingmedium 1010.

Next, big sounds are picked by the microphone 613 at a time t5,movements are detected at a time t6, and opening-closing movements ofthe door 1 are detected at times t7 and t8, respectively. For example,the child 2 sings aloud, the movements of the child 2 are detected, andthe child 1 leaves the living room from the door 1. In response, themonitoring data resulting from the third event (note that, in this case,three events have occurred) is recorded as recording data Rec3 in therecording medium 1010.

After a while, big sounds are picked by the microphone 613 at a time t9,movements are detected by the camera 611 at a time t10, the window 1 isopened at a time t11, and big sounds are picked by the microphone 613again at a time t12. For example, the child 2 sings aloud, and then thechild 2 moves to the window 1, opens the window 1, and sings aloudagain. In response, the monitoring data resulting from the fourth event(note that, in this case, four events have occurred) is recorded asrecording data Rec4 in the recording medium 1010.

Next, the child 1 enters the living room from the door 1 at a time t13,and the window 1 is closed at a time t14. In response, the monitoringdata resulting from the fifth event (note that, in this case, threeevents have occurred) is recorded as recording data Rec5 in therecording medium 1010.

As described above, when the event-related data and the monitoring dataare recorded in the recording medium 1010, if the user wishes to checkthe monitoring data, the HGW 600 can present the monitoring data to thesmartphone GUI-1 in various forms.

FIG. 3 shows an example of the internal structure of the systemcontroller 602 shown in FIG. 1. An event determination unit 625determines the detection signals from the respective sensors which havebeen described above with reference to FIG. 2. If an event is detected,a recording instruction unit 621 transmits the monitoring data 1012 tothe recording medium 1010 and instructs the recording medium 1010 torecord the monitoring data 1012. At the same time, the event-relateddata is transmitted to and recorded in the recording medium 1010.

If the event determination unit 625 receives a specific instructionsignal from the smartphone GUI-1, the event determination unit 625 alsodetermines it as an event. For example, when the first user who has thesmartphone GUI-1 is talking on the phone with the second user who staysat home, the first user can transmit an event initiation signal to theHGW 600 by operating a specific key of the smartphone GUI-1. Further,even when the first user is not talking on the phone, the first user cantransmit an event initiation signal to the HGW 600 by operating aspecific key of the smartphone GUI-1. Still further, the second user whostays at home can transmit an event initiation signal to the HGW 600 byintentionally operating a sensor. For example, to perform maintenance,the second user can transmit an event initiation signal to the HGW 600by intentionally operating a sensor which senses on-off operations of alight (for example, by covering and uncovering a light receiving unit ofthe sensor).

If the user wishes to check monitoring data, the user can request theHGW 600 (system controller 602) to play monitoring data related to adesired event via the smartphone GUI-1 or a television receiver GUI-2which is connected to the Internet.

Therefore, the system controller 602 comprises a playback control unit623 which arbitrarily outputs event-related data and monitoring datafrom the recording medium 1010. The playback control unit 623 includesan event processing unit which performs a fast-forward function, areverse function, a frame advance function, and also performs eventintegration processing. Further, since a large volume of event-relateddata and monitoring data is recorded in the recording medium 1010, thesystem controller 602 allows the user to effectively check the desiredmonitoring data.

Therefore, the system controller 602 comprises a filtering unit 631which can classify or select various events and a display styleprocessing unit 629 which can produce a display list or a display array.The produced display array or the output monitoring data are transmittedto a monitor such as the smartphone GUI-1 or the television receiverGUI-2 via a display data output unit 627. Further, the system controller602 includes a memory 624 which temporarily stores the data or the list.

The system controller 602 communicates with the smartphone GUI-1 or thetelevision receiver GUI-2 and transmits the produced display array orthe output monitoring data to the monitor. Further, according to aninstruction from the smartphone GUI-1 or the television receiver GUI-2,the playback control unit 623 can execute the fast-forward function, thereverse function, or the frame advance function on a video which recordsan event. Still further, the playback control unit 623 includes an eventprocessing unit for event-related data processing and performs eventarrangement order control processing, event selection processing, andthe like.

FIG. 4 shows a state where a menu is displayed on the screen of thesmartphone GUI-1, for example. In the menu, for example, there areselection buttons such as a monitoring data request button 501, anInternet (1) connection button 502, an Internet (2) connection button503, a phone activation button 504, a game (1) start button 505, and agame (2) start button 506. Further, there is also a sensor list button507, and when the user operates this button 507, the user can display alist of event detection sensors.

Here, it is assumed that the user operates the monitoring data requestbutton 501. Then, for example, as shown in FIG. 5A or 5B, the followingmessage: “Which event of video would you like to check?” and buttons512, 513 and 514 such as the following buttons: “Select all”, “Specify”and “Select the same as usual” are displayed on the screen of thesmartphone GUI-1.

When the user perform a touch operation and selects the button 512:“Select all”, under the control of the control data management unit 601,event occurrence time data (of all events regardless of sensors), and apart (thumbnails) of monitoring data (video data recorded by the camera)of when the respective events have occurred are transmitted to thesmartphone GUI-1. Since a large volume of event-related data andmonitoring data is recorded in the recording medium 1010, theevent-related data and the representative thumbnails of thecorresponding monitoring data of an event which has occurred, forexample, five hours ago and of a plurality of (three to five) eventswhich have occurred around that time are selected and displayed as thedisplay data corresponding to the display start time. The representativethumbnail is the monitoring data (video data) corresponding to the eventoccurrence time, for example.

When the message: “Which event of video would you like to check?” isdisplayed on the screen of the smartphone GUI-1, the user can select adesired event by performing a touch operation on the button 513:“Specify”. In this case, for example, a list 517 of the reference namesof the active sensors in the house (opening-closing movement of door 1,opening-closing movement of door 2, opening-closing movement of window1, opening-closing movement of window 2, . . . , and the like) isdisplayed. The user can touch and select one or more of events whosevideos the user wishes to check. In the case shown in FIG. 5A, theitems: “opening-closing movement of door 1”; “opening-closing movementof window 1”; and “movement detection” are selected and determined. Notethat, although only a few simple events are shown in this case, moreevents and more titles of events will be set in reality.

When the user selects events whose videos the user wishes to check andperforms a determination operation 518, as will be described later, therepresentative thumbnails of the monitoring data of when the selectedevents have occurred and the corresponding event-related data aredisplayed. In this case also, since a large volume of event-related dataand monitoring data is recorded in the recording medium 1010, theevent-related data and the representative thumbnails of thecorresponding monitoring data of an event which has occurred, forexample, five hours ago and of a plurality of (three to five) eventswhich have occurred that time are selected and displayed as the displaydata corresponding to the display start time.

When the message: “Which event of video would you like to check?” isdisplayed on the screen of the smartphone GUI-1, the user can select thesame events of videos as usual by performing a touch operation on thebutton 514: “Select the same as usual”. Note that the button 514 will beactive on the condition that the button 513: “Specify” has been operatedbefore and then the determination operation 518 has been performed. Inthis case, based on an event which has already been specified, theevent-related data and the representative thumbnails of thecorresponding monitoring data of an event which has occurred, forexample, five hours ago and of a plurality of (three to five) eventswhich have occurred around that time are selected and displayed.

In FIG. 5A, the monitoring data is separately managed from one selectedevent to another, and the event-related data of each selected event ischronologically arranged. An example of the display corresponding to thearrangement on the screen of the smartphone will be described later withreference to FIG. 8A. However, the display of the event-related data isnot limited to this example. It is possible, by making a setting shownin FIG. 5B, to display the event-related data of various events incombination (FIG. 8B).

That is, as shown in FIG. 5B, before the determination button 518 b, acombination button 518 b may be displayed. If the combination button 518a is operated, it is possible to chronologically display theevent-related data of the selected items in the event list (in thiscase, two items: opening-closing movement of door 2; and audiodetection) in combination. That is, when the combination button 518 aand the determination button 518 b are operated in succession, forexample, such an array of event-related data as that shown in FIG. 8B,which will be described later, will be displayed.

As described above, before the user requests the control data managementunit 601 to play the monitoring data related to the desired events, theuser can notify the control data management unit 601 of the events whosevideos the user wishes to play.

FIG. 6A shows an operation screen which is to be displayed after themonitoring data request button 501 is operated in the menu of thesmartphone GUI-1 shown in FIG. 4. Here, as previously described, themessage: “which event of video would you like to check?” and the buttonssuch as the buttons 512, 513 and 514: “Select all”, “Specify” and“Select the same as usual” are displayed to the user. It is assumedthat, for example, the user selects the button 514: “Select the same asusual”. Then, for example, an event list such as that shown in FIG. 6Bis displayed. As described with reference to FIG. 3, the playbackcontrol unit 623 reads event-related data and monitoring data from therecording medium 1010, the filtering unit 613 performs the filteringprocessing, and then the display style processing unit 629 produces theevent list. Note that, in the filtering processing, it is also possibleto read the event-related data from the recoding medium 1010 by theplayback control unit 623, apply the filtering processing to theevent-related data, and then output the monitoring data corresponding tothe extracted event-related data from the recording medium 1010 in thisorder.

In the above example, a case where an event list is displayed inresponse to a request from the smartphone GUI-1 has been described, butthe same also applies to a case where an event list is displayed inresponse to a request from the television receiver GUI-2. Note that, inthe case of the television receiver GUI-2, it is possible to perform anoperation by operating a cursor on the screen using a remote controller.

Further, although thumbnails of the monitoring data are simplified inFIG. 6B, images captured at the respective viewing angles of the camera611 will be displayed in an actual list.

Now, a thumbnail 522 of an event 521 is selected from the list of FIG.6B by a touch operation. Then, the playback control unit 623 (shown inFIG. 3) starts continuous playback of monitoring data of, for example,about ten minutes long since five minutes before the event 521 hadoccurred, and the monitoring data is transmitted to the monitor. Thevideo to be displayed in this example is shown in FIG. 7.

In the video shown in FIG. 7, a person 525 opens a door 526 and enters aroom, walks to a bed 527, and lies on the bad 527. When a list of aplurality of items of monitoring data corresponding to a plurality ofitems of event-related data is displayed according to an instructionfrom the smartphone GUI-1 or the television receiver GUI-2, and if adesired item of monitoring data is selected from the displayed list ofthe plurality of items of the monitoring data, the playback control unit623 continuously plays the specified monitoring data for thecorresponding recording time.

Further, according to an instruction from the smartphone GUI-1 or thetelevision receiver GUI-2, as will be described below, the playbackcontrol unit 623 (shown in FIG. 3) can perform the fast-forwardfunction, the reverse function, and the frame advance function on thevideo which records the event.

Still further, since the playback control unit 623 can refer toevent-related data, the playback control unit 623 can continuously playa plurality of items of monitoring data related to a plurality of eventsin the fast-forward mode or in the ordinary mode.

Still further, the playback control unit 623 can play a plurality ofitems of monitoring data related to a specified event in thefast-forward mode or in the ordinary mode.

Still further, the display control unit shown in FIG. 3 comprises anevent processing unit which processes a plurality of items ofevent-related data, and the event processing unit can integrate andprocess a plurality of items of event-related data corresponding to aspecific event. For example, there is a case where a sporadic eventoccurs. For example, a big sound or a movement (such as a swing of ablind) is sporadically detected. In such a case, a detection time of asporadic event should be integrated into a sequence of detection times,and monitoring data should be presented to the user based on integratedevent-related data.

In the above-described embodiment, video data which is recorded by thecamera 611 for a certain time (five, ten, fifteen or twenty minuteslong) in response to event detection is recorded as monitoring data.Here, the recording time of monitoring data of each event may beappropriately changed. Further, the recording time of one event may bedifferent from the recording time of another event. Still further, therecording time of monitoring data may vary depending on the time of dayin which an event occurs.

As the event arrangement method (method of arranging thumbnailscorresponding to event-related data), any arrangement method can beapplied by an arrangement application, and thumbnails corresponding toevent-related data will be displayed according to the arrangementmethod.

FIG. 8A shows a display example where event-related data and thumbnailsof monitoring data related to the event-related data are classified byevent category. In the example, an event issued in response to movementdetection, an event issued in response to an opening-closing movement ofdoor 1, and an event issued in response to an on-off operation of light1 are shown as event categories.

Now, the smartphone GUI-1 displays thumbnails 525 a to 525 dcorresponding to event-related data 526 a to 526 d related to eventsissued in response to opening-closing movements of the door 1. Theevent-related data 526 a to 526 d are arranged in chronological order.Here, if the user swipes the touch operation surface of the smartphoneGUI-1 in the direction of an arrow 531 a, event-related data of eventswhich have occurred at even later times and the corresponding thumbnailsare displayed, and if the user swipes the touch operation surface of thesmartphone GUI-1 in the direction of an arrow 531 b, event-related dataof events which have occurred at even earlier times and thecorresponding thumbnails are displayed.

Further, if the user swipes the touch operation surface of thesmartphone GUI-1 in the direction of an arrow 532 a, event-related datarelated to events issued in response to opening-closing movements of thewindow 2 and the corresponding thumbnails are further displayed, and ifthe user swipes the touch operation surface of the smartphone GUI-1 inthe direction of an arrow 532 b, event-related data related to eventsissued in response to events issued in response to movement detectionand the corresponding thumbnails are further displayed.

In the above-described embodiment, as described above with reference toFIGS. 3 and 5B, it is also possible to display the event-related dataand the corresponding thumbnails of the plurality of events incombination. As shown in FIG. 3, the system controller 602 includes thefiltering unit 631. The filtering unit 631 can filter and classifyevent-related data into event categories, or can combine and listevent-related data of various events. FIG. 8B shows a display example ofthe event-related data and the thumbnails of the monitoring data ofevents issued in response to opening-closing movements of the door 2 andof events issued in response to sound detection.

In this case also, if the user swipes the touch operation surface of thesmartphone GUI-1 in the direction of an arrow 531 a, event-related dataof events which have occurred at even later times and the correspondingthumbnails are displayed, and if the user swipes the touch operationsurface of the smartphone GUI-1 in the direction of an arrow 531 b,event-related data of events which have occurred at even earlier timesand the corresponding thumbnails are displayed.

FIG. 9 is an explanatory diagram showing another example of therelationship between the smartphone and the event-related data displayedon the screen of the smartphone and the operation method of theevent-related data displayed on the screen of the smartphone. In theprevious display examples (FIGS. 8A and 8B), event-related data islisted. However, after the monitoring data display is touched, thereference names of event occurrence sources may be displayed in tilestyle as shown in FIG. 9. Further, when the user taps his/her finger ona desired tile of a plurality of tiles (door 1, door 2, window 1 towindow 4, light 1 to light 5, sound 561, sound/girl 562, sound/boy 563,. . . ), the display may be changed to the display shown in FIG. 6B, forexample. If the sound 561 is selected, event-related data of eventsissued in response to all sounds will be displayed. However, if thesound/girl 562 is selected, event-related data of events issued inresponse to sounds of girls will be displayed, and if the sound/boy 563is selected, event-related data of events issued in response to soundsof boys will be displayed.

FIG. 10 shows the structure of the event-related data recorded in themanagement area 1021 and the monitoring data recorded in the data area1011. The event-related data is classified by event category. Forexample, the event categories include an opening-closing movement of adoor, an opening-closing movement of a window, an on-off operation of alight, an on-off operation of an air conditioner, an on-off operation ofa television receiver, movement detection, and the like. Further, eachevent category item includes sensor items (sensor 1, sensor 2, . . . )which are provided with sensor identification data, respectively. Stillfurther, each sensor item includes a plurality of items of event data.The event data includes, for example, an event occurrence time, arecording start time of monitoring data, a recording end time ofmonitoring data, a recording start address of monitoring data, arecording end address of monitoring data, a thumbnail address, and thelike. Here, the recording start address of monitoring data, therecording end address of monitoring data, and the thumbnail address allindicate addresses in the data region 1011, and the playback controlunit 623 reads desired data from the recording medium 1010 withreference to these addresses.

FIG. 11 shows an embodiment comprising a recording medium 1010′ in arecording playback device 1030 which is connected to the HGW 600. Therecording medium 1010, which has been described with reference to FIG.1, is provided in the server 1000 which is connected to the HGW 600 viathe Internet 300. However, in the embodiment shown in FIG. 11, therecording medium 1010′ is provided within the recording playback device1030 which is connected to the HGW 600.

In the present embodiment, for example, monitoring data can be easilydisplayed with a high degree of definition in the television receiverGUI-2. Further, since monitoring data will not be transmitted to theoutside via the Internet 300, the present embodiment is effective in thecase of managing, in particular, personal monitoring data. Note thatdata to be transmitted to the server 1000 or data to be transmitted fromthe server 1000 to the HGW 600 via the Internet 300 is subjected toconcealment processing.

Further, in the embodiment shown in FIG. 11, the camera 611, themicrophone 613, and the speaker 615 can be connected to or disconnectedfrom the HGW 600. That is, the camera 611, the microphone 613 and thespeaker 615 can be installed at sites distant from the main body of theHGW 600 via connection lines.

Still further, as another embodiment, both the recording medium 1010shown in FIG. 1 and the recording medium 1010′ shown in FIG. 11 may beprovided. In this case, the recording places for monitoring data andevent-related data may be arbitrarily selected.

For example, monitoring data and event management data of events whichare within the contractual coverage of a security service provider maybe recorded in the recording medium 1010 of the server 1000, andmonitoring data and event management data of the other events which areexcluded from the contractual coverage may be recorded in the recordingmedium 1010′ in the personal recording playback device 1030.

FIG. 12 is an example of the sensor list which is displayed on thescreen of the smartphone GUI-1 when the sensor list button 507 of thesmartphone GUI-1 is operated. The sensor list is managed by the displaystyle processing unit 629.

In the sensor list, the identification names of the respective sensorsused at home (for example, door 1, door 2, movement detection 1, windowswitch 1, . . . , and the like) are displayed. In the sensor list, acheck mark shows whether or not each sensor is in operation. That is, acheck mark: “V” is added to an item on the list: “On” if a sensor isincorporated into a system as an event detector, and a check mark: “V”is added to an item on the list: “Off” if a sensor is not incorporatedinto a system as an event detector. The user can arbitrarily set a checkmark: “V”. For example, the user can turn a desired sensor on or off byperforming a touch operation on the corresponding item on the list.

Further, the user can arbitrarily set, when each sensor detects anevent, the recording place for the event-related data of the event andthe corresponding monitoring data to the external server or thehousehold recording playback device 1030 shown in FIG. 11. According tothe setting, a check mark: “V” will be displayed in an item on the list:“Server” in the case of recording in the external server 1000, while acheck mark: “V” will be displayed, for example, in an item on the list:“Option” in the case of recording in the household recording playbackdevice 1030.

FIG. 13 shows that the HGW 600 is compatible to various communicationstandards. There is a case where communication standards of sensors varyfrom sensor manufacturer to sensor manufacturer. For example, there aresensors 101 a, 102 a, 103 a, 104 a, . . . conforming to the IEEE802.15.4 standard, sensors 101 b, 102 b, 103 b, 104 b, . . . conformingto the IEEE 802.151 standard, and sensors 101 c, 102 c, 103 c, 104 c, .. . conforming to the IEEE 802.15.3a. Further, there may be sensorsconforming to the IEEE 802.11b standard, the IEEE 802.11a standard, andthe IEEE 802.11g standard.

Therefore, the HGW 600 of the present embodiment comprises interfaces605 a, 605 b and 605 c as the network interfaces compatible to therespective communication standards.

FIGS. 14A, 14B and 14C show the HGW 600 of various types. The HGW 600includes an HGW 600 which integrally comprises the camera 611, themicrophone 613, and the speaker 615 as shown in FIG. 14A, and an HGW 600which is appropriately connectable to the camera 611, the microphone613, and the speaker 615 as shown in FIG. 14B. Further, the HGW 600includes an HGW 600 which integrally comprises the camera 611, themicrophone 613, and the speaker 615 as shown in FIG. 14A and the HGW 600is integrated with a traveling device 618. Note that, since the HGW 600can take various forms as described above, the HGW 600 may also bereferred to as an assistant device 605 as a whole.

FIG. 15A shows a sensor 800 as an event detector. For example, a switch802 is provided in a substrate 801. A blade 803 is attached to one endof the substrate 801 via a hinge. For example, when a door or a windowis opened, the rotating portion of the blade 803 is separated from thesubstrate 801, and then the switch 802 is turned on. In this way, poweris supplied from a power source to a power circuit provided in thesubstrate 801, a radio signal transmitter in the substrate 801 isactivated, and a radio signal containing a predetermined sensor ID isoutput. At this time, the switch 802 is turned on (that is, a door or awindow is open), as the radio signal is received by the HGW 600, the HGW600 recognizes that the door or the window is open. If the HGW 600 doesnot receive any radio signal containing the predetermined sensor ID, theHGW 600 recognizes that the door or the window is closed.

FIG. 15B shows a sensor 810 as another event detector. For example, aphotoelectric converter (photoelectric converter panel) 812 is attachedto the substrate 811. As the photoelectric converter 812 performs anoutput, a radio signal transmitter 813 is driven. If the photoelectricconverter 812 is not irradiated with light, the photoelectric converter812 swiftly discharges all power. Therefore, for example, if a curtainis opened or a light is turned on, a radio signal containing a sensor IDis output from the radio signal transmitter 813. In contrast, if acurtain is closed or a light is turned off, the radio signal transmitter813 stops the operation, and the radio signal output will stop,accordingly. Therefore, the sensor 810 can be used as a sensor whichdetects an opening-closing movement of a curtain, an on-off operation ofa light, or the like.

Further, it is also possible, by providing a color filter on the lightreceiving surface of the photoelectric converter 812, to configure thesensor 810 not to respond to undesirable light.

Note that, to detect an opening-closing movement of a curtain, anothersimilar sensor, namely, a second sensor may be further provided inaddition to the above-described sensor 810. Further, the second sensorturns a switch off when a curtain is closed and light is blocked by thecurtain, while the second sensor turns the switch on when the curtain isopened and the second sensor is illuminated with light, and transmits aradio signal containing the ID of the second sensor for a certain periodof time. In this case, if one sensor has a problem, the HGW 600 caneasily determine the problem. Therefore, it is possible to enhance thedetection performance of the function of detecting the opening-closingmovements of the curtain in the HGW 600 system.

In the above description, one camera is provided, and video data whichis recorded by the camera is mainly used as monitoring data. However, aplurality of cameras (camera 1, camera 2, camera 3, . . . ) may beprovided as sensors, and a plurality of items of video data which arerecorded by the plurality of cameras may be treated as monitoring data.In that case, when the user checks the monitoring data, the user mayselect which monitoring data of cameras to check. Further, monitoringdata of a plurality of cameras may be separately displayed on thescreen.

Still further, certainly, the monitoring data is not limited to a videobut also includes a change in temperature/humidity and a change inpressure. Therefore, when the user checks the monitoring data, forexample, the user may select which type of monitoring data to check in amanner similar to that shown in FIG. 9.

As a modification, the present invention may further comprise a means ofediting monitoring data as memorial data. For example, a typical happyfamily life scene or the like changes with time or with the times. In acase where such scenes are recorded as monitoring data, monitoring dataof a specific event can be extracted, edited, and stored as a memorialvideo. Therefore, memorial video storage processing can be executed withrespect to particular monitoring data.

FIG. 16 shows another embodiment. In the present embodiment, the systemcontroller 602 further comprises a function 641 of specifying aparticular recorded event, a function 642 of unspecifying a particularrecorded event, and a function 643 of playing a specified event. In thefollowing, since the other function blocks are the same as those of theprevious embodiment, the function blocks the same as those of theprevious embodiment are denoted by the same reference number, anddetailed description thereof will be omitted.

At home, for example, there is a case where a happy family life scene isseen or there is also a case where a child is born and a new familymember is added to a family. For example, a happy family life scene isseen in the dining room. Therefore, in the present embodiment, thesensor list includes a particular event specification item on the listas shown in FIG. 17.

In the example of FIG. 17, for example, a video to be recorded by amonitoring camera C2 which is installed in the dining room is designatedas an event for specific use. If a movement is detected based on videodata, the monitoring camera C2 can video a happy family life scene.Further, for example, a sound (a cry is regarded as an event) to bepicked by a microphone 2 of a child's room is designated as an event forspecific use. Therefore, for example, if a child is born in a family,events related to the baby can be recorded as memorial events of thebaby. Still further, in each of the dining room and the child's room,the camera and the microphone may be integrally operated. In this way,both the audio data and the video data of the memorial events can beautomatically recorded.

Note that the present invention is not limited to the above-describedembodiments. In response to an event, the above-described system canrecord audio data, video data, and the like in a recording medium.Therefore, in response to an event with a high degree of importance, notonly some cameras and/or microphones, but all cameras and/or microphonesmay be activated. In this way, the situations (videos and sounds) of therespective installation areas of all the cameras and microphones can berecorded as event data. Here, events with a high degree of importanceinclude, for example, an opening-closing movement of a front door, anopening-closing movement of a safe, an opening-closing movement of adrawer, and the like. About the above-mentioned technical term, variousexpressions are possible. As for the above-mentioned technical term,those names are not necessarily limited. For example, you may exchangedevice (or unit) for unit (or device).

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. A data monitoring and management devicecomprising: a network interface to obtain event-related data when atleast a sensor detects an event being received as input; a datamanagement unit to (i) control at least an operation of a camera and anoperation of a microphone and (ii) send one or both of first monitoringdata from the camera and second monitoring data from the microphone to arecording unit; a system controller controlling the data managementunit, wherein the system controller includes an instruction unit to urgethe event-related data of when the event is detected and the one or bothof the first monitoring data and the second monitoring data of when theevent occurs to be recorded in the recording unit; a filtering unitfiltering the event-related data; and a display data output unitreproducing the event-related data and a part of the monitoring data ofthe event corresponding to the event-related data, which are extractedby the filtering unit, from the recording unit and outputting the dataas display data, and the filtering unit receives conditions for thefiltering from a network.
 2. The data monitoring and management deviceof claim 1, wherein the system controller receives an instruction torequire monitoring data from the network.
 3. The data monitoring andmanagement device of claim 2, wherein the system controller receives aninstruction “to designate an event to be checked” and/or an instruction“to check an event which is checked at any time” as an option afterreceiving the instruction to require monitoring data from the network.4. The data monitoring and management device of claim 3, wherein ifreceiving the instruction “to check an event which is checked at anytime” from the network, the system controller outputs event detectiondata from a sensor corresponding to a previously checked event via thedisplay data output unit.
 5. The data monitoring and management deviceof claim 3, wherein if receiving an instruction “to check all ofevents”, after receiving the instruction to require the monitoring data,the system controller outputs event detection data from all of sensors.6. The data monitoring and management device of claim 1, wherein thesystem controller outputs display data to display each of thumbnails ofthe events via the display data output unit.
 7. The data monitoring andmanagement device of claim 1, wherein the system controller outputsdisplay data to display each of thumbnails of the events via the displaydata output unit and, if receiving an instruction to select onethumbnail, plays and outputs event monitoring data corresponding to thethumbnail, and play sections of the event monitoring data are sectionsbefore and after occurrence of the corresponding event.
 8. The datamonitoring and management device of claim 1, wherein the systemcontroller outputs display data to display each of thumbnails of theevents via the display data output unit and, a device requiring thedisplay data enables the thumbnail to be scrolled.
 9. The datamonitoring and management device of claim 1, wherein the systemcontroller outputs display data to display a list of the event-relateddata via the display data output unit and, a device requiring thedisplay data enables the event-related data to be scrolled.
 10. The datamonitoring and management device of claim 1, wherein if inputting aninstruction to require monitoring data via a network, the systemcontroller outputs display data of blocks identifying the events to thenetwork via the display data output unit.
 11. The data monitoring andmanagement device of any one of claims 1 to 10, wherein a device givingthe instruction to the system controller via the network is a mobileterminal.